Automatic detection of musicians' ancillary gestures based on video analysis

A novel approach for the detection of ancillary gestures produced by clarinetists during musical performances is presented in this paper. Ancillary gestures, also known as non-obvious or accompanist gestures are produced spontaneously by musicians during their performances and do not have meaning in sound, but they help in the creation of music. The proposed approach consists in detecting, segmenting and tracking points of interest and parts of the musician body in video scenes to further analyze if the movement associated to these points of interest or body parts could be related to ancillary gestures. In particular, we tackle the problem of detecting the three most commonly seen ancillary gestures of this class of musicians: clarinet bell moving up and down, bending of the knees and shoulder curvature. In this paper we show that the optical flux algorithm for tracking a point of interest at the bottom of the clarinet bell and the projection profile algorithm for analyzing the knees and the shoulder regions are effective in detecting ancillary movements related to the clarinet, knee movement and body curvature respectively. These techniques were evaluated with respect to the precision and recall in detecting ancillary gestures on 12,423 video frames of nine clarinetists’ presentations recorded in a studio. The experimental results have shown that the precision in detecting ancillary gestures varies between 78.4% and 92.8%, while the recall varies between 85.3% and 95.5%. These results also imply that any further analysis of the videos by specialists could focus on less than 500 frames which represents a reduction of more than 99% in the